A Pipelined Algorithm for Large, Irregular All-Gather Problems

We describe and evaluate a new, pipelined algorithm for large, irregular all-gather problems. In the irregular all-gather problem each process in a set of processes contributes individual data of possibly different size, and all processes have to collect all data from all processes. The pipelined algorithm is useful for the implementation of the MPI Allgatherv collective operation of MPI (the Message-Passing Interface) for large problems. By conception the new algorithm is well suited to implementation on clustered multiprocessors, like for instance SMP clusters. The new algorithm has been implemented within different MPI libraries. Benchmark results on NEC SX-8, Linux clusters with InfiniBand and Gigabit Ethernet, IBM Blue Gene/P, and SiCortex systems show huge performance gains in accordance with the expected behavior. This paper is a revised version of the conference presentation “A Simple, Pipelined Algorithm for Large, Irregular All-gather Problems” that appeared in Recent Advances in Parallel Virtual Machine and Message Passing Interface. 15th European PVM/MPI Users’ Group Meeting, volume 5205 of Lecture Notes in Computer Science, pages 84–93, Springer, 2008. This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357.